Previously:

1. Initialization for distribution requires the user to make application-level changes for different backends (nccl/mpi) in each distributed module (torch/mcr-dl), as shown here: https://github.com/OSU-Nowlab/MCR-DL/blob/main/benchmarks/utils.py#L61.
2. The application code becomes messy when accessing the distributed module, and it also requires exporting args or knowing args.dist every time when accessing dist, as shown below:

if args.dist == 'torch':
    import torch.distributed as dist
elif args.dist == 'mcr_dl':
    import mcr_dl as dist

Now:

1. Initialize dist using mcr_dl.init_processes() replacing torch.distributed.init_process_group() in application.
2. Access dist by dist = mcr_dl.get_distributed_engine() instead of torch.distributed as dist in application.